% Copyright 2011-2015 David Hadka.  All Rights Reserved.
%
% This file is part of the MOEA Framework User Manual.
%
% Permission is granted to copy, distribute and/or modify this document under
% the terms of the GNU Free Documentation License, Version 1.3 or any later
% version published by the Free Software Foundation; with the Invariant Section
% being the section entitled "Preface", no Front-Cover Texts, and no Back-Cover
% Texts.  A copy of the license is included in the section entitled "GNU Free
% Documentation License".

\chapter{Advanced Topics}

\section{Configuring Hypervolume Calculation}
The hypervolume calculation is an important tool when comparing the performance of MOEAs.  This section details the available configuration options for the hypervolume calculation.

The hypervolume calculation computes the volume of the space dominated between the Pareto front and the nadir point.  The nadir point is set to the extremal objective values of the reference set plus some delta.  A non-zero delta is necessary to ensure such extremal values contribute a non-zero volume.  This delta is configurable by adding the following line to \file{global.properties}:

\begin{lstlisting}[language=Plaintext]
org.moeaframework.core.indicator.hypervolume_delta = 0.01
\end{lstlisting}

The hypervolume calculation is computationally expensive.  Use of the built-in hypervolume calculator may become prohibitive on Pareto fronts with $4$ or more objectives.  For this reason, it may be beneficial to use third-party hypervolume calculators instead.  A number of researchers have released C/C++ implementations of high-performance hypervolume calculators, including those listed below.

\begin{itemize}
  \item \webpage{http://ls11-www.cs.uni-dortmund.de/rudolph/hypervolume/start}
  \item \webpage{http://iridia.ulb.ac.be/~manuel/hypervolume/}
  \item \webpage{http://www.wfg.csse.uwa.edu.au/hypervolume/}\footnote{Some source code editing is necessary to modify the input and output format to be compatible with the MOEA Framework.}
\end{itemize}

Such hypervolume calculators can be used by the MOEA Framework by following two steps.  First, download and compile the desired hypervolume code.  This should result in an executable file, such as \file{hyp.exe}.  Second, configure the MOEA Framework to use this executable by adding a line similar to the following to the \file{global.properties} file.

\begin{lstlisting}[language=Plaintext]
org.moeaframework.core.indicator.hypervolume = hyp.exe {0} {1} {2} {3}
\end{lstlisting}

This property is specifying the executable and any required arguments.  The arguments are configurable by using the appropriate variable, such as \{0\}.  The complete list of available variables are shown in the table below.

\begin{center}
\begin{tabular}{ll}
  Variable & Description \\
  \hline
  \{0\} & Number of objectives \\
  \{1\} & Approximation set size \\
  \{2\} & File containing the approximation set \\
  \{3\} & File containing the reference point \\
  \{4\} & The reference point, separated by spaces
\end{tabular}
\end{center}

If all else fails, the hypervolume calculation can be disabled.  When disabled, the hypervolume will be reported as \java{NaN}.  To disable all hypervolume calculations, add the following line to \file{global.properties}:

\begin{lstlisting}[language=Plaintext]
org.moeaframework.core.indicator.hypervolume_enabled = false
\end{lstlisting}

\section{Storing Large Datasets}
When dealing with large datasets, proper data organization and management is key to avoiding headaches.  A number of tools are provided by the MOEA Framework for storing and manipulating large datasets.  The two key classes are the \java{ResultFileWriter} and \java{ResultFileReader}.  A result file is a collection of one or more approximation sets.  Each entry in the result file is the approximation set, including the decision variables and objectives for all solutions in the approximation set, and any additional metadata you provide.  Note that this approximation set does not contain any constraints, as only feasible solutions are written in a result file.

\subsection{Writing Result Files}

The \java{ResultFileWriter} class is used to write result files.  The example code below demonstrates running the UF1 problem and recording the approximation set at each generation.  In addition, two pieces of metadata are stored: the current number of objective function evaluations (NFE) and the elapsed time.

\begin{lstlisting}[language=Java]
Problem problem = null;
Algorithm algorithm = null;
ResultFileWriter writer = null;
long startTime = System.currentTimeMillis();

try {
  problem = ProblemFactory.getInstance().getProblem("UF1");  
  algorithm = AlgorithmFactory.getInstance().getAlgorithm(
      "NSGAII", new Properties(), problem);

  try {
    writer = new ResultFileWriter(problem, new File("result.set"));

    //run the algorithm
    while (!algorithm.isTerminated() && 
        (algorithm.getNumberOfEvaluations() < 10000)) {
      algorithm.step(); //run one generation of the algorithm
    
      TypedProperties properties = new TypedProperties();
      properties.setInt("NFE", algorithm.getNumberOfEvaluations());
      properties.setLong("ElapsedTime", System.currentTimeMillis()-start);
      
      writer.append(new ResultEntry(algorithm.getResult(), properties));
    }
  } finally {
    //close the result file writer
    if (writer != null) {
      writer.close();
    }
  }
} finally {
  //close the problem to free any resources
  if (problem != null) {
    problem.close(); 
  }
}
\end{lstlisting}

\begin{important}
If the file you are saving already exists, the \java{ResultFileWriter} appends any new data to the end of the file.  If you do not want to append to any existing data, delete any old file first.
\end{important}

\subsection{Extract Information from Result Files}
The \java{ExtractData} command line utility is an extremely useful tool for extracting information from a result file.  It can extract any properties from the file as well as calculate specific performance indicators, and outputs this data in a clean, tabular format which can be read into spreadsheet software, such as LibreOffice Calc or Microsoft Excel.  When only extracting metadata, you need only specify the input file and the property keys to extract.  For instance, continuing the example from above, we can extract the \plaintext{NFE} and \plaintext{ElapsedTime} properties with the following command:

\begin{lstlisting}[language=Plaintext]
java org.moeaframework.analysis.sensitivity.ExtractData
    --problem UF1
    --input result.set
    NFE ElapsedTime
\end{lstlisting}

The output of this command will appear similar to:

\begin{lstlisting}[language=Plaintext]
#NFE ElapsedTime
100     125
200     156
300     172
400     187
500     203
...
\end{lstlisting}

Performance indicators can be calculated using one of the ``plus options.''  The options for the supported performance indicators include \plaintext{+hypervolume} for hypervolume, \plaintext{+generational} for generational distance, \plaintext{+inverted} for inverted generational distance, \plaintext{+epsilon} for additive $\epsilon$-indicator, \plaintext{+error} for maximum Pareto front error, \plaintext{+spacing} for spacing, and \plaintext{+contribution} for reference set contribution/coverage.  In addition, you must specify the problem, reference set, and optionally the $\epsilon$ values to use when calculating contribution.  For example:

\begin{lstlisting}[language=Plaintext]
java org.moeaframework.analysis.sensitivity.ExtractData
    --problem UF1
    --input result.set
    --reference ./pf/UF1.dat
    NFE ElapsedTime +hypervolume +epsilon +coverage
\end{lstlisting}

The added performance indicators will appear alongside the other properties:

\begin{lstlisting}[language=Plaintext]
#NFE ElapsedTime +hypervolume +epsilon +contribution
100     125          0.0      1.287951      0.0
200     156          0.0      1.149751      0.0
300     172          0.0      1.102796      0.0
400     187          0.0      1.083581      0.0
500     203          0.0      0.959353      0.0
...
\end{lstlisting}

Additional command line options allow you to format the output, such as removing the column header line or specifying the column separator character.

\section{Dealing with Maximized Objectives}
\label{sect:maximizing}
The MOEA Framework is setup to minimize objectives; it can not by itself maximize objectives.  This simplifies the program and increases its performance considerably.  By only allowing minimization objectives, the MOEA Framework can avoid the overhead of constantly determining the optimization direction whenever calculating the Pareto dominance relation.

This approach, however, puts the burden on the user to make the appropriate adjustments to their problem definition to allow maximization objectives.  The easiest way to allow maximization objectives is to negate the objective value, as demonstrated below:

\begin{lstlisting}[language=Java]
solution.setObjective(0, obj1);
solution.setObjective(1, -obj2); //negate the original objective value
\end{lstlisting}

By minimizing the negated objective value, we are maximizing the original objective value.  These negated objective values will be carried through to any output files produced by the MOEA Framework.  The help assist in managing these output files, version 1.13 includes the \java{Negater} command line utility.  The \java{Negater} tool processes any output file produced by the MOEA Framework and negates any specified objective.  For example, without the two objective example above, we can remove the negation in any output file with the following command.  Specifying a direction of 1 will negate the corresponding objective values in the processed file.

\begin{lstlisting}[language=Plaintext]
java org.moeaframework.analysis.sensitivity.Negater
    --direction 0,1
    output.set
\end{lstlisting}

\begin{important}
It is best to wait until all post-processing is complete before negating the objectives back to their original, maximized form as any calculations on the maximized form will be invalid.  You can always apply the \java{Negater} a second time to undo the change.  It is the responsibility of the user to manage their data files accordingly.
\end{important}


\section{Checkpointing}

The MOEA Framework provides checkpointing functionality.  As an algorithm is running, checkpoint files will be periodically saved.  The checkpoint file stores the current state of the algorithm.  If the run is interrupted, such as during a power outage, the run can be resumed at the last saved checkpoint.  The \texttt{setCheckpointFile} sets the file location for the checkpoint file, and \texttt{checkpointEveryIteration} or \texttt{setCheckpointFrequency} control how frequently the checkpoint file is saved.

Resuming a run from a checkpoint occurs automatically.  If the checkpoint file does not exist, a run starts from the beginning.  However, if the checkpoint file exists, then the run is automatically resumed at that checkpoint.  For this reason, care must be taken when using checkpoints as they can be a source of confusion for new users.  For instance, using the same checkpoint file from an unrelated run can cause unexpected behavior or an error.  For this reason, checkpoints are recommended only when solving time-consuming problems.

The code snippet below demonstrates the use of checkpointing.

\begin{lstlisting}[language=Java]
NondominatedPopulation result = new Executor()
		.withProblem("UF1")
		.withAlgorithm("NSGAII")
		.withMaxEvaluations(1000000)
		.checkpointEveryIteration()
		.withCheckpointFile(new File("UF1_NSGAII.chkpt"))
		.run();
\end{lstlisting}

\begin{important}
Checkpoint files are never deleted by the MOEA Framework.  Each time you run this example, it will resume from its last save point.  If you want to run this example from the beginning, you must delete the checkpoint file manually.  In this example, the checkpoint file is saved in the \folder{\moeaframework} folder.
\end{important}


\section{Referencing the Problem}
Once a new problem is defined in Java, it can be referenced by the MOEA Framework in a number of ways.  This section details the various methods for referencing problems.

\subsection{By Class}
The \class{Executor}, \class{Instrumenter} and \class{Analyzer} classes introduced in \chptref{chpt:executor} all accept direct references to the problem class using the \java{withProblemClass} method.  For example, following the previous example with the Kursawe problem, we can optimize this problem with the following code:

\begin{lstlisting}[language=Java]
new Executor()
		.withProblemClass(Kursawe.class)
		.withAlgorithm("NSGAII")
		.withMaxEvaluations(10000)
		.run();
\end{lstlisting}

Note how the Kursawe problem is specified by name followed by \java{.class}.  This passes a direct reference to the Kursawe class we created in the previous chapter.

Problems can also define constructors with arguments.  For example, consider a problem that needs to load data from a file.  For this to work, define a constructor in the problem class that accepts the desired inputs.  In this case, our constructor would be called \java{public ProblemWithArgument(File dataFile) { ... }}.  You can then solve this problem as shown below.

\begin{lstlisting}[language=Java]
new Executor()
		.withProblemClass(ProblemWithArgument.class, new File("inputFile.txt"))
		.withAlgorithm("NSGAII")
		.withMaxEvaluations(10000)
		.run();
\end{lstlisting}

\subsection{By Class Name}
As of version 1.11, problems can be referenced by their fully-qualified class name.  The fully-qualified class name includes the Java package in which the class is defined.  For example, the Schaffer problem's fully-qualified class name is \classpath{org.moeaframework.problem.misc.Schaffer}.  The problem \emph{must} have an empty (no argument) constructor.

The class name can be used to run problems anywhere the MOEA Framework accepts a string representation of the problem.  This includes but is not limited to
\begin{enumerate}
  \item The \java{withProblem} method in \class{Executor}, \class{Instrumenter} and \class{Analyzer}
  \item Any command line utilities with a problem argument
  \item The problem selection combo box in the MOEA Diagnostic Tool
\end{enumerate}

\subsection{By Name}

The MOEA Framework also provides the option to reference problems by name.  There are two advantages to using this approach.  First, this approach allows the use of short, meaningful names.  For example, rather than specifying the fully-qualified class name for the Schaffer problem, \classpath{org.moeaframework.problem.misc.Schaffer}, one can use the name \class{Schaffer} instead.  Second, a reference set for named problems can optionally be defined.  This reference set will be automatically used wherever a reference set is required.  Without this, a reference set must be manually specified by the user or programmer each time it is required.

The disadvantage to this approach is that some additional configuration is necessary to provide the mapping from the problem name to the problem class.  As such, this approach is recommended for third-party library developers who are developing new problems to be used with the MOEA Framework.  The remainder of this section describes two such methods for referencing problems by name.

The problem name can be used to run problems anywhere the MOEA Framework accepts a string representation of the problem.  This includes but is not limited to
\begin{enumerate}
  \item The \texttt{withProblem} method in \class{Executor}, \class{Instrumenter} and \class{Analyzer}
  \item Any command line utilities with a problem argument
  \item The problem selection combo box in the MOEA Diagnostic Tool
\end{enumerate}

\subsection{With a ProblemProvider}
The first way to reference problems by name is to define a \class{ProblemProvider}.  The \class{ProblemProvider} uses the Java Service Provider Interface (SPI).  The SPI allows the MOEA Framework to load all available providers from the classpath.  This approach allows third-party software vendors to distribute compiled JAR files containing \class{ProblemProvider} instances that are automatically loaded by the MOEA Framework.  To create a new \class{ProblemProvider}, first create a subclass of the \class{ProblemProvider} class.  To do so, you must define two methods:
\begin{enumerate}
  \item \java{Problem getProblem(String name)}
  \item \java{NondominatedPopulation getReferenceSet(String name)}
\end{enumerate}

Both methods are provided the problem name as the argument.  The \texttt{getProblem} method should return a new instance of the specified problem, or \java{null} if the provider does not support the given problem name.  Likewise, the \java{getReferenceSet} method should return the reference set of the specified problem if one is available, or \java{null} otherwise.  Returning \java{null} when the problem is not supported by the provider is important, as the Java SPI will scan all available \class{ProblemProvider} instances until it finds a suitable provider.

\begin{lstlisting}[language=Java]
import org.moeaframework.core.NondominatedPopulation;
import org.moeaframework.core.Problem;
import org.moeaframework.core.spi.ProblemProvider;

public class ExampleProblemProvider extends ProblemProvider {

	public Problem getProblem(String name) {
		if (name.equalsIgnoreCase("kursawe")) {
			return new Kursawe();
		} else {
			return null;
		}
	}

	public NondominatedPopulation getReferenceSet(String name) {
		return null;
	}

}
\end{lstlisting}

Lastly, a special configuration file used by the SPI must be created.  The file is located at \file{META-INF/services/org.moeaframework.core.spi.ProblemProvider}.  Each line of this file must contain the fully-qualified class name for each of the \class{ProblemProvider}s being introduced.  When bundling the compiled class files into a JAR, be sure that this configuration file is also copied into the JAR.

Once packaged as a JAR, the provider is ready to be used.  Place the JAR on the classpath used by the MOEA Framework.  Once on the classpath, the Java SPI mechanism used by the MOEA Framework will be able to scan and load all providers contained in all available JAR files.

\subsection{With the \texttt{global.properties} File}
The second way to reference problems by name is to add the problem definition to the \file{global.properties} file.  This \file{global.properties} file contains the configuration options for the MOEA Framework.  This file usually accompanies a MOEA Framework distribution, but in the event it does not exist, you can just create a new empty text file.  Adding a new problem is as simple as adding the following two lines to \file{global.properties}:

\begin{lstlisting}[language=Plaintext]
org.moeaframework.problem.problems = Kursawe
org.moeaframework.problem.Kursawe.class = Kursawe
\end{lstlisting}

Line 1 lists all problems configured in the \file{global.properties} file.  The string provided here becomes the problem name.  This is the name you would subsequently provide to any of the MOEA Framework tools to instantiate the problem.  More than one problem can be specified by separating the problem names with commas.

Line 2 identifies the class for the specified problem.  Note that this entry follows the naming convention \plaintext{org.moeaframework.problem.NAME.class = value}.  The \plaintext{NAME} used must match the problem name defined in line 1.  The value is the fully-qualified Java classname.  In this case, the class is located in the default package.  If this class were located, for example, in the package \classpath{foo.bar}, the value must be set to \class{foo.bar.Kursawe}.

The reference set file for the problem can be optionally specified as well.  If a reference set is available for the problem, add the following line to \file{global.properties}:

\begin{lstlisting}[language=Plaintext]
org.moeaframework.problem.Kursawe.referenceSet = kursawe.ref
\end{lstlisting}